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f. Introduction 

Traditionally, the motivation for studying Hamiltonian systems comes either 
from classical mechanics or from partial differential equations. In this paper we 
study a particular class of Hamiltonian systems which are motivated in a completely 
different way: they arise from dynamical systems associated to game theory. This 
class of Hamiltonian systems was proposed in [T^] and corresponds to Hamiltoni- 
ans which are piecewise affine. In this paper we study several properties of such 
dynamical systems. 

For general smooth functions H : K 2n — > K for which is (topologically) a 

In — 1 sphere, very little is known about the global dynamics of the corresponding 
Hamiltonian vector field, even if we make convexity assumptions on the level sets. 
If H is a quadratic function, the system is completely integrable and the flow is 
extremely simple. So instead, in this paper we consider Hamiltonians H which are 
piecewise affine and investigate whether the following analogy holds: 



circle diffeomorphism 
quadratic map x H y 1 — ax 2 
Hcnon map (x,y) i— > (f — ax 2 - 
Smooth Hamiltonian 
Smooth area preserving maps 



by,x) 



circle rotation 

tent map x H ► 1 — a\x\ 

Lozi map (x, y) \-¥ (1 — a\x\ + by, x) 

Piecewise affine Hamiltonian 

Piecewise affine area preserving maps 



This analogy would suggest that one might gain insight about smooth Hamiltonian 
systems by looking at piecewise affine ones, in the same way as circle diffcomor- 
phisms can be modelled by circle rotations. Much of the complexity of the dynamics 
may well persist in the piecewise affine case, and even though these systems are not 
smooth they still may be easier to study. 



The authors would like to thank Abed Bounemoura and Vassili Gelfreich for several valuable 
comments. 
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This paper consists of two parts. In the first part we prove a classification 
theorem for the case when n — 2 which goes towards a description of the global 
dynamics in terms of coding, while in the second part we give numerical results 
which show that much richness of the dynamics in the smooth case persists. 

p2n 



Let us be more precise. We say that H : 



is a piecewise affine function 



if 



,Z k in 



so that H is 



(i) H is continuous and 

(ii) there exists a finite number of hyper-planes Z\ , 

affine on each component of K 2 ™ \ U** =1 Zi. 

„. ■ ■ m fdH dH\ . . . , , , 

bmce H is piecewise afhne, tt~> — are piecewise constant outside the hyper- 

\dq dp J 

planes Zi and there the derivatives are multivalued. Consider the Hamiltonian 
system 

dp dH dq dH 
( ' dt £ ~dq~ , dt e ~~dp~ 7 

or more generally the Hamiltonian vector field Xh associated to H and the symplcc- 
tic 2-form y\ ■ u^dpt A dqj (where (wy) are the (real) coefficients of some constant 
non-singular n x n matrix f2) and the corresponding differential inclusion 



(1.2) 



(Here Xh is defined by requiring that oj(Xh, Y) = dH(Y) for each vector field Y.) 
In other words, 



(1.3) 



( * \ 

dt 

dq 
\dt J 







n 
o 



( d -\ 

dp 

dH 
V ~dq J 



where f2' stands for the transpose of the matrix f2. The reason we write G rather 

OH dH 

than = in the above equations is because — — , — — are multivalued. In this gen- 

dq dp 

erality, there is no reason to assume that the flow of these differential inclusions is 
continuous. 



Instead we will consider the special case of H : E x E 



defined by 



(1.4) 



H{p, q) = max (A4q)i — min (p M. )j 



where p' stands for the transpose of p, (Ai q)i and (p' M. )j stand for the i-th 
and j-th component of the vectors A4 q respectively p' Ai and E consists of the 
set of probability vectors in M". Here we consider the corresponding Hamiltonian 
systerro defined on the space E x E. To ensure that level sets are simple, we make 
the following assumption on A4 : there exist p, q € E so that all its components 
are strictly positive and so that p' M. = Al' and Mq = fil for some A, /i 6 R where 
1 = (11 ... 1) G R™. A motivation for this assumption and an interpretation of 



^Differential equations with discontinuities along a hypcrplane are often called 'Filippov sys- 
tems', and there is a large literature on such systems, see for example [4], [6] and [7]- The special 
feature of the systems we consider here is that they have discontinuities along n- (n— 1) intersecting 
hyperplanes in S X S. 
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p, q is given in the paragraph above equation (|1.6|) . As was shown in 12 , one then 
has the following properties: 

(1) For an open set of full Lebesgue measure of n x n matrices A4 , each level set 
H^ 1 (g), g > is topologically a (2n — 3)-dimensional sphere and if _1 (0) = 
{(p, q)} (note that dim(E x E) = 2n — 2). In fact, H^ 1 (g) bounds a convex 
ball. 

(2) For an open set of full Lebesgue measure of n x n matrices f2 and M , 
there exists a unique solution of (|1.2j) for all initial conditions and the 
corresponding flow (p,q,t) <-> <frt(p,q) is continuous, which is not generally 
true for differential inclusions. 

(3) The flow (p, q, t) 4>t{p, q) is piecewise a translation flow, and first return 
maps to hyperplanes in _ff _1 (p) are piecewise affine maps. 

In the second part of this paper we will present some numerical studies of examples 
of such systems when n = 3, i.e., when each energy level H~ 1 (g), g > 0, is topolog- 
ically a three-sphere. Let us single out one of these examples: there exists an open 
set of 3 x 3 matrices Ad and fi with the following property. For each g > there 
exists a topological disc D C H^ 1 (g) which is made up of 4 triangles in E x E so 
that 

(1) the first return map R to D is continuous and extends continuously to the 
closure of D; 

(2) R\dD = id (in fact 3D corresponds to a periodic orbit of the flow, whose 
Floquet multipliers are undefined); 

(3) R: D — > D is area-preserving and piecewise affine; 

(4) each orbit in H^ 1 (g) intersects D infinitely often. 

Moreover, R: D — >• D contains hyperbolic horseshoes and also an elliptic periodic 
orbit, surrounded by an elliptical disk consisting of quasi-periodic orbits on invariant 
ellipses, see [HJEI]. Numerical simulations for these systems suggest that all orbits 
outside this elliptical disk in D are dense in this set, see Example 2 in Part 2 of this 
paper. This first return map could be a piecewise affine model for general smooth 
area preserving maps of the disk. 

As mentioned, one motivation for looking at Hamiltonians as in (|1.4[) and the 
corresponding systems (|1.2|) comes from game theory. Indeed, these correspond 
to certain differential inclusions which are naturally associated to so-called zero- 
sum games in game theory. For this, let A, B be n x n matrices and define 
BR-A(q) '■= argmaXpgjjp'Aq 1 and BIZb{p) '■— argmiriges p'Bq and consider the dif- 
ferential inclusion: 

(1.5) f t eBJl A (q)-p,f t eBJl B ( P )-q. 

The zeros of this equation, i.e. the set of points (p, q) for which p e BJZa(q),Q S 
BJZb(p), are called Nash equilibria. The differential inclusion (II .5[) is called the Best 
Response Dynamics and the corresponding time-rescaled 

| G \{m A {q) -p), ^ G \{BR. B {p) - q), 

the Fictitious Play Dynamics associated to the game with matrices A and B. In 
game theory and economics these are often used to model learning, i.e. to describe 
how people 'learn' to play a game. It turns out that in the case of zero-sum games, 
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i.e. when A = —B, all orbits of (|1.5|) converge to the set of Nash equilibria, see [9] 
and for a short proof see [5]. 

Note that if we define A4 := A = —B then the existence of p, q £ £ for which 
all coordinates are strictly positive and for which p' M. = \V and Ai q = fil (as 
assumed just below (|1.4|l ) implies that £ BRa{q) ~P and £ BJZb(p) — q- So such 
a point (p, q) £ £ x £ is a Nash equilibrium. To see this, notice that BR-m (q) is 
the convex hull of all the unit vectors corresponding to the largest component (s) of 
M q. So if all components of p, q are strictly positive, then £ BJZa(<i) — p holds iff 
all coordinates of M. q are equal. 

We should note that p and q have a different connotation here from the usual 
one in classical mechanics: p corresponds to the position (the probablity vector 
describing past play) of the first player and q the corresponding object for the 
second player. Although the equation (jl.5p itself is not Hamiltonian at all, it is 
closely related to the Hamiltonian system (|1.2j) . Indeed, for (p,q) £ £ x £ \ {p, q}, 
let l(p,q) be the half-line from (p, q) containing (p,q) and define 

(1.6) ir:ZxX\{p,q}^H-\l), 

where 7r(p, q) £ i/ _1 (l) is the intersection of l(p,q) with the 2n — 3-dimensional 
sphere i/ _1 (l). It turns out that the projection of the flow on i/ _1 (l) corresponds 
to the solution of a Hamiltonian system as above. In other words, the Hamiltonian 
dynamics describes the 'spherical coordinates'. So we will think of the dynamics of 
(|1.5|) as inducing Hamiltonian dynamics. For more details see Section [2] 

In this paper we will study Hamiltonian dynamics coming from Best Response 
dynamics as in (jl.5p in the case where n = 3. The aim of this paper is the following: 

• Because of the special nature of the Hamiltonian systems we consider in 
this paper, one can associate in a natural way itineraries to each orbit. 
In the Main Theorem of this paper we will show that not all itineraries 
are possible and we will give a full classification of all possible transition 
diagram^. 

• The Hamiltonian dynamics appearing in this paper is much simpler than 
usual: the first return maps to planes are piecewise translations. In the 
second part of this paper we will show numerical simulations concerning a 
number of examples of such systems. The four examples which we show 
here, display (conjecturally): 

(1) fully ergodic behaviour; 

(2) elliptic behaviour of a very simple type (which we can prove rigorously, 
see [12]); 

(3) elliptic behaviour of a 'composite' type; 

(4) Arnol'd diffusion, and intertwining of various elliptic regions. 

• We believe that the elliptic behaviour occuring in our systems satisfies a 
huge amount of regularity, and we will formulate several questions and 
conjectures formalizing this. 

Part 1. Combinatorial results 

In the first part of the paper we introduce the Best Response and Fictitious 
Play Dynamics inducing a special case of the Hamiltonian Dynamics presented 



A transition diagram shows all allowable transitions within orbits (but not all allowable tran- 
sitions correspond to actual orbits). 
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above. We introduce a combinatorial description of the BR dynamics and provide 
a combinatorial characterisation of the dynamics for zero-sum games for n = 3 
(inducing Hamiltonian dynamics with two degrees of freedom). 

2. Hamiltonian and Best Response Dynamics 

We define XU to be the simplex of probability row vectors in R™ and E^ the 
simplex of probability column vectors in R n : 

- [p G R lxn : Pl > 0, = l} ! E B = { 9 £ r ixl : Ql > = *} ■ 

We denote E = E^ x E B . 

We consider a bimatrix (A, S), where A = (aij),B = (bij) G M nx ". With a 
slight abuse of notation we identify the standard unit vector with the integer fc 
and define the following correspondences^: 



ERa{<i) = argmaxpAg = argmax (Aq) i for g G E^, 
BRs(p) — axgm&xpBq = argmax (pB) . for p G E^- 

These correspondences are singlevalued almost everywhere, except on a finite 
number of hyperplanes. On these hyperplanes (also called indifference planes) at 
least one of the ^.correspondences has as its values the set of convex combinations 
of two (ore more) unit vectors. 

We can now define the Fictitious Play Dynamics as a continuous time dynamical 
system in E: 

(FP) at 1 

^e- t (BJZ B (p(t))- q (t)), 

for t > 1 and some given initial value (p(l),g(l)) G E. Note that the right hand 
side is singlevalued almost everywhere. 

Although (|FPp is more common in game theory, where it serves as a model of 
myopic learning, we prefer to consider the following time reparametrisation, referred 
to as the Best Response Dynamics: 

^€BJl A (q(s))-p(s), 
(BR) ds 

-lem B (p(s))- q (s). 

The orbits of both systems coincide (differing only in time parametrisation) but 
(IBRp has the advantage of being autonomous. The existence of solutions for any 
initial conditions follows from upper semicontinuity of BIZa and BIZb , see [TJ Chapter 
2.1]. 



''in a game theoretic context the matrices are payoff matrices of two players A and B, where 
each player has n pure strategies, i.e. rows or columns, to choose from. and Eg are the spaces 
of mixed strategies, i.e. probability distributions over the n pure strategies that are represented 
as the standard unit vectors. BIZa and BR-b are the two players' respective best-response cor- 
respondences, assigning a payoff-maximising pure strategy answer to the strategy played by the 
opponent. 
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The classical learning processes (jFP[) and (|BRj) are naturally related to Hamil- 
tonian Dynamics via the following construction. Let H : Y,a x -> M be defined 

as 

(2.1) H(p,q) = max (AqY - min (pA) , . 

i j ■> 

Further let £1 = (c<%) be some non-singular n x n matrix. Consider a Hamiltonian 
vector field Xh associated to H and the symplectic 2-form Y)^ oJijdpi A dqj. The 
corresponding differential inclusion is 

M (f, 

Let us denote 2TU = T£ B = {w G W 1 : J2 v i = 0}, and let fl',A' be the transpose 
of the matrices Q, and A. Further let P A :R n -> TZ A ,P B : R" TS B be the 
parallel projections to TS^ and TE^ along the vectors f2' -1 T and i7 — 1 1_ respectively 
(where 1 = (11 ... 1) G M"). Then a simple calculation (see [T2]) shows that the 
above inclusion (12.21) takes the form 



(2.3) 



at 

J G PbQ-^SKbCp). 



The projections Pa,Pb appear in these equations because H is considered as a 
function on S = T*a xEb and so the dynamics is constrained to this affine subspace. 
The Hamiltonian differential inclusion (|2.3j) is closely related to the BR dynamics 
if we take ft — A. In this case the dynamics defined on iJ _1 (l) by (|2.3[) equals the 
BR-dynamics projected via tt (defined in equation (jl.6jl ) to this level set of H (for 
details see [H]). In other words, if we compute an orbit under the BR-dynamics 
then the image under tt of this orbit is an orbit under the Hamiltonian dynamics 

o. 

For this reason, for the rest of this paper we will assume ft — A, where A G M 3x3 
is non-singular. 

For later use, let us make some simple observations. can be divided into 
n convex regions Rf = BJZ^ 1 (ei), where i G {l,...,n} and analogously Rf = 

m^{ ej )cX A ,j e{i,...,n}. 

Since BIZa x BRb is constant on R l3 = R? x Rf, (|FP|) and (jBRj) have con- 
tinuous orbits which are piecewise straight lines heading for vertices (efe,e;) G £ 
whenever (p(t), q(t)) G Ru- The orbits only change direction at a finite number of 
hyperplanes, namely whenever BRa or BRb (or both) become multivalued. More 
precisely, p(t) changes direction whenever q(t) passes from Rf to Rf for some 
i ^ i' , and vice versa. See FiglUfor an example with n = 3. 

3. Combinatorial Description for the Case n = 3 

In this section and later on we restrict our attention to the case of dimension 
n = 3. The partition of E into the convex blocks ijy quite naturally gives rise to 
coding of orbits of (jBRI) and (|FP|) . We codify an orbit (p(t),q(t)) by a (finite or 
infinite, one-sided) itinerary (ioijo) - > {ik,jk) —>■... indicating 

that there exists a sequence of times (tk), such that (p(t),q(t)) G -Ri fcJfc for < 
t < tk+i- To simplify notation we will often write instead of Rij. 
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Figure 1. An example of Ha,^b C R 3 being partitioned into 
regions Rf and Rf. The dashed lines indicate the hyperplanes at 
which one of the BR correspondences is multivalued. Also shown is 
a piece of an orbit with initial conditions (po j <7o ) ■ The orbit changes 
direction four times: when p crosses a dashed line, q changes direc- 
tion, and vice versa. Note that this is an orbit for the BR dynamics 
on £ = Y>a x Eg, not for the induced Hamiltonian dynamics on 
H-\l). 




Figure 2. A hypothetical graph representing transitions for the 
itineraries of orbits of BR dynamics. We will see later that this 
very graph cannot be a transition graph for a bimatrix game. 



Abstractly we have a graph of nine vertices i,j — 1,2,3 with directed 

edges between them (see FigJ2]). 

It is not difficult to see that for an orbit of BR dynamics for a fixed bimatrix 
(A,B), such itinerary can then contain — > i ^ i' (or j ^ f ) if and 

only if di>j > dij (biji > 6y). Note that for almost all initial conditions in the cor- 
responding orbits p and q never switch directions simultaneously, i.e. the itinerary 
only contains transitions of the form — ► and — > (hj')i * 3 j' 

(see[Il]). 
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Figure 3. (a) example of a transition diagram 

(b) dominated row 

(c) dominated column 

(d) ,(e) examples of alternating cycles (which will be shown to be 
impossible in zero-sum games) 



Further we want to make sure that for any i i' (J ^ j') there is only one 
possible transition direction between and {{hj) and Therefore 

we introduce the following non-degeneracy assumption on the bimatrix (A, B): 

Assumption 1. fly ^ a^j and bij ^ bij> for all with i ^ i' and j ^ j'. 

Clearly the set of bimatrices satisfying this assumption is open dense with full 
Lebesgue measure in the space of bimatrices. 

The possible transitions can be expressed in a transition diagram as in FigJHJa). 
The three rows and three columns of the diagram represent the regions Rf, i = 
1, 2, 3 and , j = 1, 2, 3, respectively. The arrows indicate the possible transitions 
between the regions, which by Assumption [T] always have a unique direction. For 
example, the itinerary of an orbit of the BR dynamics for a given bimatrix can 
contain (1,2) —> (1,3) if and only if in the first row of its transition diagram an 
arrow points from the second into the third column. Opposite sides of the diagram 
should be thought of as identified, so that possible transitions between the first and 
third rows and columns are indicated by arrows on the boundary of the diagram. 
It is important to note that this partition does not have the nice properties of a 
Markov partition: there is no claim that every itinerary that can be obtained from 
the transition diagram can actually be realised by an orbit of the BR dynamics. 

One may now ask whether a given transition diagram is realisable as the transi- 
tion diagram of BR dynamics for a bimatrix game (A, B), and how properties of a 
game relate to the combinatorial information given by its transition diagram. 

A simple first observation is that no row of a transition diagram can have three 
horizonal arrows pointing in the same direction, as this would imply a^ j > a^f > 
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Oijii > at j. Analogously no column of such diagram can have three vertical arrows 
pointing in the same direction. 

It is easy to see that apart from this restriction, any transition diagram can 
be realised by appropriate choice of (A, B). However, our interest lies in bimatrix 
games whose BR dynamics displays 'non-trivial' behaviour. For this we introduce 
the following assumption: 

Assumption 2. In the transition diagram no row (column) is dominated by an- 
other row (column), i.e. no three vertical (horizontal) arrows between two rows 
(columns) point in the same direction (Fig\B(b)-(c)) 0. Formally, let {i,i',i"} = 
{jd'J"} = {1,2,3}, then 

-> and (i,f) -> (i',f) => j") 

and 

-> (i,f) and -> (£',/) => (i",f) -> (i", j). 

4. Zero-Sum Games 

In game theory, an important class of (bimatrix) games are the zero-sum games, 
i.e. games (A,B), such that A + B = 0. However, our analysis remains valid for a 
larger class, namely games that are linearly equivalent to a zero-sum game. 

Definition 1. Two 3x3 bimatrix games (A,B) and (C,D) are linearly equiva- 
lent, if there exist e > 0, fj,j = 1,2,3 and g > 0, hi, i = 1, 2, 3 such that 

Cij = easy + fj and <!,, = gh, , + //,. 

It can be checked that for linearly equivalent bimatrix games (A, B) and (C, D), 
the respective best-response correspondences coincide: BJZa — BRc and BJZb = 
BRd- From the definitions it immediately follows that linearly equivalent bimatrix 
games induce the same dynamics (|FPj) and (|BR[) . Since our main focus lies on 
these dynamical processes, in the rest of this text we call a game (A, B) zero-sum, 
if there exists a linearly equivalent game (C, D), such that C + D = 0. 

Definition 2. A point (p,q) € £ is called Nash Equilibrium if p £ BR,A(q) and 
qeBJZ B (p). 

By the definition, Nash Equilibria are precisely the fixed points of (|FP|) and (jBRjl . 
It has been proved by John Nash, that every game with finitely many players and 
strategies has got a Nash Equilibrium (see [S]). A very important classical result is 
the following: 

Theorem 1 (Brown [3], Robinson [5]). For a zero-sum game, every orbit of the 
KBR\ ) and \FP\l converges to the set of Nash Equilibria. 

A short proof using an explicitly given Lyapunov function can be found in [5] . In 
the same paper, Hofbauer states the converse conjecture, which still remains open: 

Conjecture 1 (Hofbauer [5]). A bimatrix game with a unique Nash Equilibrium 
point in S that is stable under the BR (or FP) dynamics must be a zero-sum game. 

%n game theoretic terms this means that none of the players has any 'strictly dominated pure 
strategy'. The existence of such strategy would mean that the dynamics essentially reduces to the 
FP/BR dynamics of a 2 X 3 game, which under mild genericity conditions is known to converge 
to the set of Nash Equilibria in a rather simple way, see [2]. 
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As the above indicates, zero-sum games are of great interest in the study of BR 
and FP dynamics. A natural question to ask is now, which combinatorial configu- 
rations (transition diagrams) can be realised by zero-sum games. In this paper we 
will restrict our attention to zero-sum games with a unique Nash Equilibrium in 
the interior of S: 

Assumption 3. The bimatrix game (A, B) has a unique Nash Equilibrium point 
(E A , E B ) , which lies in the interior of S. Equivalently (see for instance [12] ). 
there exists precisely one point E B £ Eb, such that (AE B )i = (AE B )j Vi,j, and 
precisely one point E A € Ea ; such that (E A B)i — (E A B)j \/i,j. 

Lemma 1. For every bimatrix game (A,B), Assumption^ implies Assumption^ 

Proof. If Assumption [2] does not hold, there is a row (column) in the transition 
diagram for (A,B) which is dominated by another row (column), say i dominated 
by i'. It follows that R A (R B ) is empty. 

On the other hand, Assumption [3] implies that for all k, I, R k i = R B x R A has 
non-empty intersection with every neighbourhood of the Nash Equilibrium. □ 

For later use, we make the following definitions: 

Definition 3. A sequence (io,jo), (ii,ji), ■ • ■ > (*n,Jn) = (jo, jo) is called alternat- 
ing cycle for (A,B), if after reversing either all downward and upward pointing 
or all right and left pointing arrows in the transition diagram of (A, B), it forms a 
directed loop. Examples of alternating cycles are shown in Fig\^d) and (e). More 
formally, either 

• (ik,jk) -> (4+1, jfe+i) whenever i k ^ i k+1 and jk+i) -> (ik,jk) when- 
ever j k ^ j k+1 , or 

• (ik+i, jk+i) -> (ik,jk) whenever i k ^ i k+1 and(i k ,j k ) (i k+1 ,j k+1 ) when- 
ever j k ^ j k+1 . 

Definition 4. We call (i,j) a sink, if it can be entered but not left by trajectories 
of the BR dynamics (i.e. (i',j) — > {i,j) and (i,j') — > (i,j) for i' ^ i and j' ^ j). 
Conversely we call it a source, if it can be left but not entered. 

We can now formulate several consequences for the transition diagram of a zero- 
sum game (A, B): 

Lemma 2. Let (A, B) be zero-sum and satisfy Assumptions]^ and\^ Then: 

(1) The transition diagram does not have alternating cycles. 

(2) The transition diagram does not have sinks. 

(3) The transition diagram does not have sources. 

Remark 1. One can see that in (1) without loss of generality we can only consider 
cycles in which the i- and j-component change alternatingly, which justifies the 
notion of alternating cycle. In fact in 3 x 3 games (1) reduces to saying that there 
are no alternating cycles of the two kinds depicted in Fig\^d) and (e). 

Proof. For statement (1) assume that A + B = (otherwise choose linearly equiva- 
lent matrices such that this holds and note that this does not change the arrows in 
the transition diagram). Now note first that (i,j) — > {i',j) iff a^j > dy. Further, 
(i,j) — > (i,j') iff bij' > bij, which is equivalent to > a,y. It follows that an 
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Figure 4. Proof of statement (3) of Lemma[2l 

(a) General case for a diagram with a source 

(b) Case 1: sink in (2,1) contradicts zero-sum 

(c) Case 2: alternating cycle contradicts zero-sum 

(d) Case 3 

(e) ,(f) Case 3: necessarily following configuration 



2 2 




M 



Figure 5. Proof of statement (3) of Lemma [2) 
This configuration necessarily follows from the transition diagram 
in Case 3 of the proof. (N, M) is then a Nash Equilibrium, con- 
tradicting Assumption [3] 
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alternating cycle leads to a chain of inequalities a,i j > at l j 1 > . . . > di n j n = a; 0j0 
and therefore cannot exist. 

To prove statement (2), we use Theorem [TJ It follows from this theorem and 
Assumption [3] that orbits of (|BR[) converge to a single isolated point in the interior 
of S. A sink in the transition diagram however would imply that orbits of (|BR| 
that start in R^ (for some stay in it for all times. Since orbits do not change 
direction while in R^j, this can only be the case if they converge (in straight line 
segments) towards a vertex on dT,. 

At last, to show statement (3) of the lemma let us assume for a contradiction that 
such a zero-sum game satisfying the assumptions and with a source in its transition 
diagram exists. After possibly permuting rows and columns and swapping the 
roles of the two players we can assume that the source is (2, 2) and we have the 
(incomplete) diagram as seen in FigHJa). Let us now consider all four possible 
cases for the vertical arrows in (2, 3): 

• Case 1: (2,3) — !• («,3), i = 1,2, i.e. both arrows pointing out of (2,3): 
Either row 2 dominates row 1 or 3, or (2, 1) is a sink, see FigfJJb). 

• Case 2: (i,3) — > (2,3), i = 1,2, i.e. both arrows pointing into (2,3): 
Either column 2 dominates column 3 or there is an alternating cycle, see 
FigHc). 

• Case 3: (3, 3) — > (2, 3), (2, 3) — > (1, 3), both arrows point upward (FigfJJd)): 
In order to avoid an alternating cycle and a dominated column, one neces- 
sarily has (3,2) — > (3,3) and (1,3) — >■ (1,2). Further, since row 2 may not 
dominate row 1 and alternating cycles cannot happen in a zero-sum game, 
one gets (1,1) -)• (2,1) and (1,2) -> (1,1), also (2,1) -> (3,1) is neces- 
sary to avoid a source in (2, 1), see FigfJJe). With some further deductions 
of the same kind one can show that the only possible transition diagram 
is the one shown in FigfJJf). We can now deduce that and are 
partitioned into the regions Rf and R® as shown in FigEl Consider the 
point on <9E denoted by (N, M) and note that BJZa(M) cotains ei and e3, 
hence all their convex combinations. Therefore N 6 BJZa(M). Analogously, 
M 6 ERb{N). Therefore (N,M) is a Nash Equilibrium contradicting our 
assumption that the interior Nash Equilibrium is unique. 

(In fact it also follows from this configuration that there exist initial 
conditions arbitrarily close to the interior Nash Equilibrium whose trajec- 
tories spiral off towards (N, M) and therefore the interior Nash Equilibrium 
cannot be stable for the dynamics.) 

• Case 4: (1, 3) — > (2, 3) and (2, 3) — > (3, 3), i.e. both arrows point downward: 
This case is analogous to the previous one. 

To conclude, we have shown that a source in the diagram contradicts our assump- 
tion of a zero-sum game with unique interior Nash Equilibrium, which finishes the 
proof of statement (3). 

□ 

5. Main Result 

The next hope is of course to get a full characterization of all combinatorial 
configurations that can be realised by zero-sum games. This indeed can be done 
after defining a suitable notion of combinatorially equivalent games. 
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Definition 5. We call two bimatrix games (A,B) and (C,D) combinatorially 
identical, if they induce the same transition relation, i.e. (i,j) —> for (A,B) 

iff(i,j)^(i',j')for(C,D). 

We call two bimatrix games (A, B) and (C, D) combinatorially equivalent, if 
there exist permutation matrices P, Q such that (A, B) and (PCQ, PDQ) are com- 
binatorially identical or (B',A') and (PCQ, PDQ) are combinatorially identical. 

The definition expresses the idea, that games are combinatorially equivalent if 
they have the same transition diagram up to permutation of rows and columns and 
transposition. The main result is the following: 

Theorem 2 (Combinatorial classification of transition diagrams for zero-sum games) 
The types of transition diagram ( combinatorial equivalence classes ) that can be re- 
alised by a zero-sum game satisfying Assumptions]]^ and [3] are precisely all those 
that satisfy the following (combinatorial) conditions: 

(1) No row of the diagram has three horizonal arrows pointing in the same 
direction and no column has three vertical arrows pointing in the same 
direction. 

(2) No three horizontal arrows between two columns point in the same direction 
and no three vertical arrows between two rows point in the same direction. 

(3) The diagram has no sinks. 

(4) The diagram has no sources. 

(5) The diagram has no alternating cycles. 

This gives precisely 23 different types of transition diagrams (up to combinatorial 
equivalence). These are listed in AvvendixV^. 

Throughout the proof of the theorem we will make use of the following notion: 

Definition 6. We call an oriented loop of length four (formed by the arrows in the 
diagram) a short loop, see Fig\6j(a). A short loop always has the form (i,j) — > 
(i',j) —> — > (i±j r ) ~ ► {hi) an d we indicate the vertex in the diagram encircled 
by such loop by a »|j 

Proof. By the above discussion we know that (1) is true for any transition diagram 
of a game and (2) corresponds to Assumption [5] (which is implied by Assumption 
|3l), so the only conditions that are left to check are (3)-(5). By Lemma [2] we 
already know that (3)- (5) are necessary conditions for a diagram to be realisable 
by a zero-sum game. 

To show that (l)-(5) are also sufficient, we will proceed in two steps: we will show 
that combinatorially these conditions give rise to precisely 23 types of diagrams 
(up to permutation of rows and columns and transposition) and then we will give 
examples of zero-sum games realising these types. Because of the initially large 
number of possible transition diagrams, we will group them by the number of short 
loops contained in them. 

Let us introduce the notion of rows (or columns) coinciding or differing at a 
position. Two rows, say i and i' , coincide at a position, say between columns j and 

^Coincidentally (or not?) the number 23 is the most sacred number for the religion called 
'Discordianism'. In this religion 23 is the number of the highest deity, Eris, who is the Greek 
goddess of Chaos. 

^It can be checked that a short loop precisely corresponds to those 2x2 subgamcs, which are 
linearly equivalent to a zero-sum game. 
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(a) 



(b) 



(c) 



\ 1 i 



(d) (e) (f) 

Figure 6. Proof of Theorem [2] 

(a) a short loop 

(b) two rows coinciding at all positions 

(c) two rows coinciding at one and differing at two positions 

(d) Lemma 0Jl): if two rows differ at two positions, then either 
there is a short loop or one row dominates the other 

(e) Lemma[5] two short loops not between the same rows/columns 

(f) Lemma|4l2) applied to columns 1 ,2 and 2,3 in previous diagram 















-t- 







j' , if (i, j) — > if and only if (i',j) — > and they differ at this position 

otherwise. E.g. in Figj6jb) rows 1 and 2 coincide at all positions, whereas in 
FigEJc) they coincide at one and differ at two positions. 

We now introduce a few very helpful lemmas about the transition diagrams of 
zero-sum games satisfying our assumptions: 

Lemma 3. Two columns (or rows) can have at most two short loops between them. 

Lemma 4. (1) // two rows (columns) differ at two positions, then there is a 
short loop between these rows (columns). 

(2) // two rows (columns) differ at all three positions, then there are precisely 
two short loops between them. 

(3) If two rows (columns) coincide at two positions, then there is a short loop 
between each of these rows (columns) and the third row (column). In par- 
ticular the diagram has at least two short loop. 

(4) If two rows (columns) coincide at all three positions, then there are precisely 
two short loops between each of these rows (columns) and the third row 
(column). Then the diagram has precisely four short loops. 
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Proof. For (1), note that every 2x2 block obtained by deleting one row and one 
column from a transition diagram has got either two arrows pointing in the same 
direction or contains and alternating cycle or a short loop. Assume that two rows 
(columns) differ at two positions (FigJHfc)). Since we don't allow alternating cycles, 
the only way a short loop between the two rows (columns) can be avoided is by 
having all arrows between them pointing in the same direction (Fig|6jd)). But this 
case is ruled out by hypothesis (2) of the theorem. Hence there is a short loop 
between them. 

Essentially the same argument shows that statement (2) of the lemma holds. 

If two rows (columns) coincide at two positions, then since no column (row) is 
allowed to be dominated, each of these rows differs at two positions from the third 
row (column). Statement (3) then follows from statement (1). 

The same argument proves statement (4). The fact that the diagram then has 
precisely four short loops follows from Lemma [3] and the fact that there cannot be 
any short loops between the two rows (columns) that coincide at all positions. □ 

We can now proceed to grouping all possible transition diagrams by the number 
of short loops contained in them: 

Lemma 5. The transition diagram of a game as in Theorem [H can only have 
between three and six short loops. 

Proof. Note first that for any pair of rows (or columns) of a transition diagram at 
least one of the cases of Lemma U applies and we can make the following list of 
cases for two rows (columns), say i and j: 

• i and j coincide at positions, then they have precisely 2 short loops 
between them. 

• i and j coincide at 1 position, then they have 1 or 2 short loops between 
them. 

• i and j coincide at 2 positions, then there is at most 1 short loop between 
them, and there is at least 1 short loop between each of them and the third 
row (column). 

• i and j coincide at 3 positions, then they have no short loops between them, 
and there are precisely 2 short loops between each of them and the third 
row (column). 

It is clear from these that there cannot be a diagram without any short loops: pick 
any two rows and whichever of the above cases applies, it follows that the diagram 
has at least one short loop. 

Similarly, the transition diagram cannot have precisely one short loop. Assume 
for contradiction that (without loss of generality) there is a single short loop between 
rows 1 and 2. Now consider rows 2 and 3, which do not have a short loop between 
them. Then they must have at least 2 coinciding positions. But having 2 or more 
coinciding positions implies that there is also a short loop between rows 3 and 1, 
which contradicts the assumption. 

Now let us show that there is no transition diagram with precisely two short 
loops. Assume first, such diagram exists and both short loops are between the 
same two rows (or columns), say rows 1 and 2. Applying the above rules to rows 2 
and 3 we see that either there have to be more short loops between rows 2 and 3 
or between rows 3 and I, both a contradiction. So the only left possibility is that 
the two short loops are neither between the same two rows nor columns. 
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Here there are two cases to check: either both short loops run clockwise or one 
run clockwise and one runs anti-clockwise (any other configuration leads to a com- 
binatorially equivalent diagram) . Assume the short loops have different orientation. 
Without loss of generality we have the configuration shown in FigJBJe) . By Lemma 
1[2) applied to columns 1,2 and 2,3 we get that (1, 1) -)■ (2, 1) and (2, 3) -)■ (3, 3) 
(FigEJf)). But now Lemma HJl) applied to columns 1,3 implies that there is a 
third short loop. A similar chain of deductions shows that the case with both short 
loops having the same orientation also cannot happen. Hence a transition diagram 
with two short loops is not possible. 

At last, the upper bound of six short loops follows directly from Lemma [3l which 
finishes the proof of the lemma. □ 

We can now state the final lemma of the proof of the main theorem. The proof of 
the lemma consists of easy (but somewhat tedious) deductions of the only possible 
combinatorial configurations for the transition diagrams and we do not provide 
complete details. Lemma [4] is very useful to reduce the number of diagrams that 
have to be checked. 

Lemma 6. Up to combinatorial equivalence, there are precisely 

• two non- equivalent transition diagrams with precisely three short loops, 

• fifteen non-equivalent transition diagrams with precisely four short loops, 

• five non- equivalent transition diagrams with precisely five short loops, 

• one transition diagram with precisely six short loops, 

that satisfy conditions (l)-(5) of the main theorem. 

Proof. Up to combinatorial equivalence, there are three ways in which three short 
loops can be positioned, see Fig[7][a)-(c). It can be checked that only the first of 
these can give a transition diagram that satisfies (l)-(5), and there are two non- 
equivalent such diagrams. 

Further, there are four ways to position four short loops (Fig|7]Jd)-(g)), the first 
three of which admit five non-equivalent transition diagrams each, whereas the last 
one contradicts (l)-(5). 

The two ways to position five short loops (Fig{7^h)- (i) ) admit two and three 
non-equivalent transition diagrams satisfying (l)-(5). At last, applying Lemma[3]it 
is obvious that up to combinatorial equivalence the only way to position six short 
loops is the one shown in FigJTfj) and it is straightforward to check that there is 
only one possible transition diagram of this type. □ 

Together with Lemma [5l this shows that there are precisely 23 transition dia- 
gram types satisfying (l)-(5). A list of the 23 diagram types and zero-sum game 
bimatrices realising them can be found in Appendix \^ which finishes the proof of 
Theorem H □ 

6. Quasi-Periodic Orbits 

In this last section of the analytic part of the paper we introduce the game- 
theoretic notion of quasi-periodicity and investigate the relation to its usual math- 
ematical definition. This notion was first introduced in |10) : 

Definition 7. We say that an orbit of BR dynamics is quasi-periodic (in the 

game-theoretic sense), if its itinerary is periodic. 
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(a) (b) (c) 



(d) (e) (f) (g) 



(h) (i) (j) 

Figure 7. The possible (non-equivalent) configurations for a tran- 
sition diagram containing 3 (a-c), 4 (d-g), 5 (h-i) or 6 (j) short 
loops 

Note that a priori this notion of quasi-periodicity of an orbit is different from 
the usual mathematical definition (of an orbit which is dense in an invariant torus). 
However, we will show that the two notions are closely related. 

We consider the Hamiltonian dynamics on S 3 corresponding to a zero-sum game 
and its first return maps to the (two-dimensional) planes on which either p(t) or 
q(t) changes direction, i.e. where one of the £>7?rCorrespondences is multivalued. 

Let S be such a plane, let x € S have a quasi-periodic orbit with (infinite) 
periodic itinerary / = I{x), and let T be the first return map to S (defined on the 
non-empty subset of S of points whose orbits return to S). Note that T acts as a 
shift by a finite number of symbols on the itinerary of x. In particular there exists 
n > 1, such that T n (x) has the same itinerary as x. Let us denote T — T n so that 
each point in the T-orbit of x has the same periodic itinerary I. 

Lemma 7. Let U = {z G S: I(z) — I(x) = 1} be the set of points in S with 
itinerary I . Then U is convex and T(U) = U . Moreover the restriction of T 
to U, T: U — >• U , is affine. 

Proof. Convexity follows from the convexity of the surfaces on which pit) or q(i) 
change direction and the fact that the flow in each region Rij follows the 'rays' of a 
central projection. By the definition of T we have that T(U) C U. We know (|12j) 
that T is area-preserving and piecewise affine on S, so that T(U) = U. Since all 
points in U have the same itinerary, it follows that T ; U — > U is indeed affine. □ 

Theorem 3. Letx £ S correspond to a quasi-periodic orbit (in the game theoretical 
sense) of the Hamiltonian dynamics of a 3 x 3 zero-sum bimatrix game, where S 
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is an indifference plane and T is the return map to S, such that I(T(x)) = I(x). 
Then one of the following holds: 

(1) The orbit of x is periodic and T n (x) = x for some n > 1. 

(2) The T -orbit of x lies on a T -invariant circle (more generally, depending on 
coordinates: an ellipse) and x corresponds to a quasi-invariant orbit of the 
Hamiltonian dynamics (in the usual sense), i.e. its orbit under the flow is 
dense in an invariant torus. 

In the second case U = {z e S : I(z) = I(x)} is a disk and T: U — > U is a rotation 
by an irrational angle (in suitable linear coordinates). Therefore in this case (2) 
holds for every z G U . 

Proof. Assume that x is not periodic. We already know that T : U — > U is a planar 
affine transformation of U. Since it is also an isometry and T(U) — U, the only 
possibility is that T restricted to U is a rotation by an irrational angle (any other 
kind of planar affine transformation satisfying these conditions would have x as a 
periodic point). The result immediately follows. □ 

The theorem shows that every quasi-periodic orbit (in the game-theoretic sense) 
is actually quasi-periodic in the usual sense. Conversely, every quasi-periodic orbit 
in the usual sense, which lies on a torus that only intersects the indifference surfaces 
along whole circles (and never just partially along an arc) is clearly quasi-periodic 
in the game theoretic sense. Throughout the rest of this paper, we will always refer 
to the game-theoretic definition, when using the notion of quasi-periodicity. 

From the argument above we can also immediately conclude: 

Corollary 1. If a Hamiltonian system induced by a 3 x 3 zero-sum bimatrix game 
has got an orbit with periodic itinerary, then it also has an actual periodic orbit. 

Part 2. Numerical Investigations and Conjectures 

In the second part of this paper we investigate the Hamiltonian dynamics induced 
by the BR dynamics numerically. We explore which types of orbit occur in these 
systems and how the combinatorial description given in the first part relates to 
these observations. 

7. Numerical Observations 

In this section we present some of our observations on the behaviour of BR 
dynamics for zero-sum games, mostly obtained from numerical experiments. The 
aspects we investigate are: 

• The time fraction that different orbits spend in each of the regions Rij. 

• The frequencies with which different orbits visit the regions and the 
transition probabilities for transitions between regions. 

• The different types of orbits that can occur and their itineraries (periodic, 
quasi-periodic, space- filling) . 

The systems we consider are randomly generated examples of zero-sum games of 
different combinatorial types, for which we look at the induced Hamiltonian dynam- 
ics on level sets of H . For randomly chosen initial points we compute the orbits of 
the BR dynamics (more precisely, its induced Hamiltonian analogue) and study the 
time fractions spent in each region Rij and the frequencies, with which the orbits 
visit the regions. Especially with respect to the presented types of orbit we do not 
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claim to give an exhaustive account of occuring types but rather a list of examples 
illustrating a few key concepts. 

Formally, for an orbit of the BR dynamics (p(t),q(t)), t > with itinerary 
(*0)Jo) (UjJi) (ik,jk) — >• ... and switching times (t n ) we define 

l n JO 

where Xij is the characteristic function of the region R^. 

Alternatively we record the number of times, that each region is being visited 
by an orbit and compute the frequencies: 

n ( \ 1 V^r u a\ v. t !■ ■ \ J 1 ' i{ (hjk) = 
Qij(n) = - > Iij(ik,3k) , where hj(ik,3k) = < n ^ 

n 10, otherwise. 

We write P BR = (/'//"'), _ ; and Q = {Q lJ ) hJ . 

Moreover throughout the following examples we look at orbits of the first return 
maps for the BR dynamics to certain surfaces of section. A convenient choice of such 
surface is a hyperplane on which either p(t) or q(t) changes direction, i.e. where one 
of the ^/^correspondences is multivalued. We will mostly use the surfaces where 
q(t) changes direction: 

S ij ={(p,q)eX:{i,j}cBR s (p)}. 
Let us now look at some examples: 

Example 1 (Uniquely ergodic case). Let the zero-sum bimatrix game (A, B) be 
given by 

/ 22 34 -4\ 
A = [ 7 -32 16 , B = -A . 
\-53 96 23 / 

We numerically calculate orbits with itineraries of 10 4 transitions for several 
hundreds of randomly chosen initial conditions. For all of these orbits, the evolution 
of P BR {n) and Q(n) indicates a convergence to 

/13 5 27\ /12 9 19^ 

P BR w 10" 2 x 14 5 27 ,Q«10- 2 x 9 13 15 
\3 1 5/ \10 5 

In FiglHl the evolution of some of the P BR (n) and Qij(n) along an orbit is 
shown. This or very similar statistical behaviour is observed for all sampled initial 
conditions. It seems to suggest that initial conditions with quasi-periodic orbits 
have zero or very small Lebesgue measure in the phase space of BR dynamics for 
this bimatrix game, as quasi-periodicity in all our experiments leads to very rapid 
convergence to certain frequencies. Most of the space seems to be filled with orbits 
that statistically resemble each other in the sense that they all visit certain portions 
of the space (the regions Rij) with asymptotically equal (or very close) frequencies. 
The same seems to hold for the fraction of time spent in each region by the orbits. 

FigJS] shows the intersections of an orbit of (|BR[) for this game with , £23 
and 5*31 (the hypersurfaces where q(t) changes direction), i.e. the orbit of the 
first return map to these surfaces. Each Su> consists of three triangular pieces, 
corresponding to the three pieces of hypersurface between regions and (i',j) 
for j — 1,2,3. Inside each of these triangles, the orbit seems to rather uniformly 
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Figure 8. Example Q] 

The first plot shows the evolution of P2\ R { n ) and Q%i{n) along 
an orbit with itinerary of length 3 x 10 4 . A certain 'stabilisation' 
and convergence to the values P21 ~ 0.14 and 921 ~ 0.09 or small 
intervals containing these values can be observed. 
The second plot shows the evolution of P^ R (n) and Q32(n) along 
the same orbit. Here the observed limits (or limit intervals) are 
near the values P32 ~ 0.01 and ~ 0.05. 

fill the space, suggesting ergodicity (of Lebesgue measure). If the BR dynamics 
had invariant tori, these would appear on all or some of these sections as invariant 
circles whose interior cannot be entered by orbits starting outside. Judging from 
the above observations, in this example they either don't exist or have very small 
radius. 

Example 2 (Space decomposed into ergodic and elliptic regions). In this example 
we consider a bimatrix game, which is an element in a family of bimatrix games 
thoroughly studied in [TT] and [13]. Let the zero-sum bimatrix game (A, B) be 
given by 

1 a\ 

a 1 , B = -A , 
a l) 

where a = v ^~ 1 ss 0.618 is the golden mean. 

Two types of orbit can be (numerically) observed for the BR dynamics of this 
game. The first type resembles the orbits in the previous example. The empirical 
frequencies P^ R (n) and Qij(n) along such orbits initially behave erratically but 
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0.3 0.32 0.34 0.36 0.38 



0.3 0.32 0.34 0.36 0.38 



0.3 0.32 0.34 0.36 0.38 



Figure 9. Example [TJ Typical orbit of induced Hamiltonian dy- 
namics intersected with the three surfaces Si 2, S23 and S31 (hy- 
perplanes, where q(t) changes direction). The original orbit has an 
itinerary of length 2 x 10 6 . The three triangular regions in each 
of the Su> correspond to the different possible transitions between 
regions and for j = 1,2,3 and are indicated by dashed 

lines. (In other words the images show the orbit of the first return 
map to the surfaces of section S«'.) 

The different visiting frequencies of the regions can clearly be seen 
by the different densities of orbit points. The orbit points inside 
each triangular region seem to be uniformly distributed. 



seem to suggest convergence to certain values or narrow ranges of values, which are 
the same for all such orbits: 

/9 11 13\ /ll 13 9\ 

P BR « 1CT 2 x 13 9 ll),QwlO~ 2 x 9 11 13 1. 

\11 13 9/ \13 9 11/ 

It can be observed that the values of Q{n) are perhaps less erratic and in most 
of our experiments they seem to converge faster than those of P BR (n). As an 
example, the evolution of P^ R (n) and Qz2{n) along a typical orbit can be seen in 

Fignni 

As in the previous example, in Fig llll we show the intersection of one such orbit 
with the surfaces Su> . Once again the orbit points have a certain seemingly uniform 
density inside each region, but here they leave out an elliptical region on each of the 
hypersurfaces. This invariant region consists of invariant circles, formed by quasi- 
periodic orbits of the system (the second type of observed orbits). The center of 
the circles corresponds to an actual periodic orbit, i.e. an elliptic fixed point of the 
return map to one of these surfaces. See |13] for an explicit analytic investigation 
of this (which is made possible by the high symmetry of this particular bimatrix 
game). 

The invariant circles in the elliptical region correspond to invariant tori in the 
BR dynamics. Their itinerary is periodic with period 6: 

(1,1) (1,2) (2,2) -> (2,3) -> (3,3) (3,1) (1,1) 

Fig ll2f a) shows the transition diagram for this bimatrix game. The periodic 
itinerary is indicated by a dashed line as a loop in the transition diagram. The 
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Figure 10. Example [U The evolution of P^ R {n) and Q 32 ( 
along a typical orbit outside of the invariant torus. 



IBM 



035 0.4 0.45 0.5 0.55 0.6 0.65 



^0 



' ..-Ir f-;- '; 



035 0.4 0.45 0.5 0.55 0.3 0.65 




0.35 0.4 0.45 0.5 0.55 6.6 



Figure 11. Example [2J Typical orbit of induced Hamiltonian dy- 
namics intersected with the three hyperplanes, where q(t) changes 
direction. The original orbit has an itinerary of length 10 6 . The 
different visiting frequencies of the regions can clearly be seen by 
the different densities of orbit points. The orbit points inside each 
triangular region seem to be uniformly distributed but leave out 
elliptical regions in some of the regions. These are filled with quasi- 
periodic orbits forming invariant circles. 



empirical frequencies along such quasi-periodic orbits converge to 



P BR = Q = 



FigfTT] suggests that there are no other invariant tori for this system, i.e. no 
open set of initial conditions outside of the visible elliptical regions, whose orbits 
are all quasi-periodic. 

A question that arises naturally from the above example is the following: does 
an invariant torus of quasi-periodic orbits always have a 'simple' periodic itinerary 
as the above? Are the periods of such elliptic islands necessarily equal to 6? As 
the next example shows, the situation can indeed be more complicated and less 
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Figure 12. (a) Example [21 The transition diagram of the bi- 
matrix game. The periodic itinerary of the quasi-periodic orbits 
forming an invariant torus is indicated by a dashed line, 
(b) Example |3j The transition diagram is the same as in the previ- 
ous example. However, there is an invariant torus of quasi-periodic 
orbits with a more complicated periodic itinerary. 



simple paths through the transition diagram are possible candidates for the periodic 
itinerary of quasi-periodic orbits in an invariant torus. 

Example 3 (Quasi-periodic behaviour with itineraries of higher period). Consider 
the bimatrix game (A, B) with 

/ 84 -37 10 \ 
A = 24 33 -14 , B = -A . 
\-26 9 20 / 

Generally, the observations here coincide with Example [T] However, one can 
detect a (quite thin) invariant torus. Fig |13l shows a typical orbit stochastically 
filling most of the space. In the bottom row of the same figure, the regions marked 
by rectangles are enlarged to reveal a thin invariant torus. These quasi-periodic 
orbits intersect S\2 once, S23 and S31 three times each. The orbits look essentially 
like the quasi-periodic orbits in the previous example, but with an extra loop added. 
The itineraries are periodic with period 13, where each period is of the form 

(1, 1) (1, 2) (2, 2) (2, 3) -> (3, 3) (3, 1) 

(1, 1) (1, 2) (3, 2) (2, 2) (2, 3) -> (3, 3) -> (3, 1) (1, 1). 

In Fig U2f b). this itinerary is shown as a loop in the transition diagram. The 
example demonstrates that combinatorially more complicated quasi-periodic orbits 
are possible for open sets of initial conditions in the BR dynamics of zero-sum 
games. 

The next example shows an even more complex quasi-periodic structure and 
gives numerical evidence for more subtle and involved effects than those observed 
above: 
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Figure 13. Example [2 Two orbits (their intersections with the 
surfaces Sa') are shown: one which stochastically fills most of the 
space and one which lies in an invariant torus. The latter intersects 
the first surface once and the other two three times each. The 
intersections of the (very thin) torus with the surfaces are marked 
with rectangles and shown magnified in the bottom row. 

Example 4 (Coexistence of different elliptic behaviour). Let us now consider the 
bimatrix game (A, B) with 

/-92 18 52 \ 
A = 62 -37 -33 , B = -A . 
\-10 9 -18/ 

As in all the previous examples, the largest part of the phase space of the BR 
dynamics seems to be filled with orbits which stochastically fill most of the space 
and along which the frequency distributions P BR (n) and Q(n) seem to converge 
to certain (orbit-independent) values. Again, an invariant torus can be found. 
It is more complicated than those observed in the other examples (see Fig fT4]l . 
The orbits forming this invariant torus are quasi-periodic and have an itinerary of 
period 60. Its structure suggests a generalisation of the type of itinerary observed 
in Example [3] It consists of a sequence of blocks of the following two forms: 

a = ((1, 1) -> (3, 1) -»• (2, 1) -> (2, 2) -> (3, 2) -> (3, 3) -> (1, 3) -> (1, 1)) and 

b = ((1, 1) -> (3, 1) -> (2, 1) (2, 2) -> (3, 2) -> (3, 3) -> (1, 3) -> (1, 2) -> (1, 1)) . 

The two blocks are shown as pathes in the transition diagram in Fig ll5l Each 
period of the itinerary of orbits in the invariant torus then has the form 

a— >a— >a— > b ^ a — > b. 
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Figure 14. Example 2) The invariant torus (corresponding to 
quasi-periodic orbits) intersects two of the hypersurfaces where 
q(t) changes direction as shown on the left, while not intersecting 
the third hypersurface at all. The right image is a magnification 
of the region marked by a rectangle. 



1 

if 



FIGURE 15. Example HJ The two block types a and b of the 
itinerary for the quasi-periodic orbits in this system. Note that 
they differ by one step only. 



As in the previous example the two blocks are the same except for one element (in 
the previous example the itinerary consisted of two blocks concatenated alternat- 
ingly). 
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Figure 16. Example [4] The left image shows a typical orbit that 
seems to fill almost all of the space. The right image shows a 
magnified view of the 'heart-shaped' region spared out by this or- 
bit. Also both images show the invariant circles of a quasi-periodic 
orbit inside the 'heart-shaped' region. 

In FigfT6]the intersection of an orbit outside of the invariant torus with one of 
the hypersurfaces is shown together with a quasi-periodic orbit. The orbit seems to 
have essentially the same property of filling the space outside the invariant torus, 
as in the previous examples. However, a closer look at a neighbourhood of the 
invariant circles (see the right part of Fig fTB)) reveals that the orbit not only misses 
out the invariant circles but also a certain 'heart-shaped' region surrounding these. 
The investigation of orbits with initial conditions in this set shows a range of effects 
not observed in any of the previous examples. 

Several different orbits with initial conditions in this region can be seen in Fig ll7l 
The orbit points show complicated structures, revealing a large number of 'stochas- 
tic' regions as well as invariant regions of periodic orbits of high periods and corre- 
sponding quasi-periodic orbits (Fig |18l shows some examples of such quasi-periodic 
orbits of different higher periods). Some of these orbits spend very long times 
(itineraries of length 10 6 and more) in the heart-shaped region before diffusing into 
the much larger 'stochastic' rest of the space. On the other hand we observe orbits 
that stochastically fill (heart-shaped) annuli leaving out islands of quasi-periodic 
orbits. These annuli seem to be invariant for the dynamics (see Fig fT^j) . 

Altogether the observations described above strongly indicate the occurrence of 
'Arnol'd diffusion': the coexistence of a family of invariant annuli, which contain 
regions of stochastic (space-filling) motion and islands of further periodic orbits and 
invariant circles (quasi-periodic orbits). 

8. Conclusion and Discussion 

We would like to propose some open questions for further investigation. 

(1) Does the Hamiltonian system induced by a 3 x 3 zero-sum bimatrix game 
always have quasi-periodic orbits / invariant tori? Example Q] suggests that 
it is possible to have topological mixing and that the Lebesgue measure 
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Figure 17. Example [1J Two different orbits with initial condi- 
tions in the 'heart-shaped' region. Regions of stochastic motion as 
well as invariant islands of periodic and qnasi-periodic orbits are 
clearly visible. The first orbit spends a long time in the 'heart- 
shaped' region before it diffuses into the larger 'ergodic' part of 
the space. 
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Figure 18. Example H) Two different quasi-periodic orbits (in- 
variant tori for the BR dynamics) of different periods. 
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Figure 19. Example [4] An orbit restricted to an invariant heart- 
shaped annulus. 

is ergodic. However, this might be due to the limited resolution of our 
numerical simulations and images. 

(2) Are there orbits which are dense outside of the elliptic regions? Are almost 
all orbits outside of the elliptic regions dense? 

(3) Example HI suggests that the system has infinitely many elliptic islands 
corresponding to quasi-periodic orbits of different periods. The pictures of 
orbits (e.g. FigflTland Fig. [T9|) show many regions that could potentially 
contain such elliptic islands of quasi-periodic orbits of different periods. 
All regions that we investigated for this property actually revealed quasi- 
periodic orbits. 

(4) Given a specific bimatrix example, are there a finite number of blocks, so 
that the itinerary of any orbit on an elliptic island is periodic with each 
period being a (finite) concatenation of these blocks? The examples we 
looked at suggest the answer to be positive. 

In Part 1 of this paper we assigned to a Hamiltonian system a transition diagram, 
giving a necessary condition on the itinerary of orbits. If we were able to develop 
some kind of 'admissibility condition' (i.e. a sufficiency condition), we could perhaps 
obtain results such as 'density of periodic orbits', in the same way as was done for 
quadratic maps of the interval. 

In Part 2 we demonstrated that this class of dynamics is sufficiently rich to 
mimic many of the intricacies of smooth Hamiltonian systems. In spite of the 
many open questions that still remain, it appears that these piecewise affine Hamil- 
tonian systems could provide a new way of gaining insight into global dynamics of 
Hamiltonian systems. 
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Appendix A. The 23 transition diagram types 

We list all 23 transition diagram types as in Theorem [2] sorted by the number of 
short loops contained in them, together with the respective matrices A, such that 
the bimatrix games (A, —A) realise the given diagram types. 
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